Direct nonlinear acceleration

Optimization acceleration techniques such as momentum play a key role in state-of-the-art machine learning algorithms. Recently, generic vector sequence extrapolation techniques, regularized nonlinear (RNA) of Scieur et al. [22], were proposed and shown to accelerate fixed point iterations. In contrast RNA which computes coefficients by (approximately) setting the gradient objective function zero at extrapolated point, we propose more direct approach, call (DNA). DNA, aim minimize (an approximation of) value instead. We adopt approach with regularizers designed prevent model from entering region functional is less precise. While computational cost DNA comparable that RNA, our significantly outperforms on both synthetic real-world datasets. focus this paper convex problems, obtain very encouraging results accelerating training neural networks.